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Most limits on the fourth generation heavy top quark (the t') are based on the assumed dominance 
of t' — > Wb, which is expected to be case in the minimal fourth generation framework with a single 
Higgs (the so called SM4). Here we show, within a variant of a Two Higgs Doublet Model with four 
generations of fermions, that, in general, a different t' detection strategy is required if the physics 
that underlies the new heavy fermionic degrees of freedom goes beyond the "naive" SM4. We find 
that the recent CMS lower bounds: m t i < 450 GeV in the semi-leptonic channel pp — > t't' — > £ubqqb 
and m t i < 557 GeV in the dilepton channel pp — > t't' — > l + l~vvbb, that were obtained using the 
customary (SM4-driven) detection strategies, do not apply. 

In particular, we demonstrate that if the decay t! — > ht dominates, then applying the "standard" 
CMS search tools leads to a considerably relaxed lower bound: m t / ^ 350 GeV. We, therefore, 
suggest an alternative search strategy that is more sensitive to beyond SM4 dynamics of the fourth 
generation fermions. 



I. INTRODUCTION 

The origin of the ElectroWeak Symmetry Breaking (EWSB), one of the most studied theoretical puzzles in particle 
physics, is yet unknown. It is anticipated that the LHC, that has already began its historic mission of collecting data 
at 7 TeV collision energy, will provide insight into the heart of this problem. In the Standard Model (SM), EWSB 
is triggered by the Higgs mechanism with a single fundamental scalar, leaving its mass unprotected from radiative 
corrections at the cutoff scale. This is known as the hierarchy problem: the tension between the cutoff scale (Planck 
scale or GUT scale) and the seemingly unnatural ElectroWeak (EW) scale, which is usually interpreted as evidence 
for new physics at the TeV scale. 

Hints for the existence of a Higgs boson have been recently reported at the LHC: ATLAS finds an excess of events 
around M/j = 126 GeV with a local significance of 3.5 standard deviations PQ, while CMS sees a smaller excess with 
a local significance of 3.1 sigma around Mh = 124 GeV [2J. Later, it was shown at the Tevatron [3] that there is a 
possibility of an excess of events corresponding to Mh between 115 GeV and 135 GeV. It is, therefore, tempting to 
consider a Higgs mass in the range (120 — 130) GeV. The LHC with higher integrated luminosity and 8 TeV cm. 
energy should unveil more precise details about the Higgs boson. 

In the past few years, several tensions between experiments and the SM predictions were observed in flavor physics 
[4"lfl"2"]. It was noted, that adding a fourth sequential generation (for reviews see [T3HH3) to the SM (the so called SM4 
which in some instances will be referred to as the "naive" four generations model) might not only be able to resolve 
these tensions [T?l - f2"5] . but may also address the hierarchy problem [3~0T[52] . In particular, when the new fermion states 
are heavy enough, their Yukawa couplings are driven to a Landau pole or a fixed point, possibly at the TeV scale, 
where some form of strong dynamics might be responsible for dynamically breaking the EW symmetry. Besides, the 
additional CP- violating phases in the presence of a fourth family [33] may be useful for Baryogenesis 34436] . 

The fact that adding a heavy fourth generation to the SM implies new physics at O (TeV) , serves as a strong hint 
that the SM4 is not an appropriate framework to account for the expected (possibly strong) dynamics in the presence 
of these new heavy fermionic states. For this purpose and others, models that go beyond the SM4 (BSM4) were 
introduced as an alternative to the naive SM4 framework [37U49] , 

One way to incorporate the low-energy effects of such possible TeV-scale strong dynamics in an acceptable phe- 
nomenological approach is multi-Higgs scenarios, which may naturally account for the composite scalars associated 
with the new strong dynamics, see e.g., [37H44] . An example of a model designed for this purpose, the 4G2HDM (a 
model with 2 Higgs doublets and a fourth family of fermions), was proposed in |43) . and was shown to agree with 
Electroweak Precision Data (EWPD) in a wide range of its parameter space. An alternative interesting approach to 
the compositeness picture is warped extra-dimension scenarios, see e.g., [2S1I3B]. Other examples of BSM4 scenarios 
were suggested in [48-51]. 

Recently, assuming the SM4 framework, CMS reported [521 155] a 450 GeV lower limit on the t' mass in the 
semileptonic channel (pp — > t't' — > [W + ] hadronic b [W~] leptonic b — > ivbqqb), and a 557 GeV lower limit in the dilepton 

channel (pp — > t't' — > [W + ] leptonic b[W~] leptonic b — > l + l~vvbb), replacing the weaker Tevatron limits [Ml [55]. The 
most recent lower bounds on the b' mass are 480 GeV [53] (Atlas) and 611 GeV [57] (CMS). In addition, the single 
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FIG. 1: Mfu distribution for the SM4 2W + 2b^ lubbqq signature (blue) and for the 4G2HDM &W+2b signature (red), for a 
set of 7 TeV LHC events with J Ldt — 1 fb _1 . For both signatures m t i — 450 GeV is assumed. The peak of the distribution 
of Mfu for the SM4 signature is around m t <, while for the new signature the peak is shifted to a significantly lower value 
coinciding with the peak of the it background. 



Higgs of the SM4 is excluded now by CMS in the range 120 — 600 GeV [55). We argue, that these seemingly stringent 
limits apply only to the SM4 framework, which is disfavored for reasons that are outlined above. In particular, in 
BSM4 frameworks, collider signals of fourth generation fermions and/or Higgs particles may be drastically altered, 
leading to weaker bounds. An example of this was discussed in 59., where an extended Higgs sector (with 4th 
generation fermions) may highly suppress the signal for Higgs production, circumventing the current limits on the 
SM4 Higgs mass. 

The purpose of this work is, therefore, to illustrate that, in general, the limits on the t' mass can be considerably 
weaker with respect to the SM4 case. We demonstrate that in the 4G2HDM framework of [43] . where new decay 
patterns of t' emerge through the scalar sector: e.g., t 1 — > ht, t' —> H + b. We thus propose an alternative search 
method, more general and, therefore, more suitable to searches for BSM4 scenarios. 



II. t' DETECTION AT THE LHC- BEYOND SM4 



As mentioned above, CMS has recently reported [52] a 557 GeV lower bound on the t' mass, replacing the previous 
limits by CDF [54] (358 GeV) and DO [55] (285 GeV) [68]. A summary on the Tevatron limits can be found at [69] . 
These searches assume Br it' — > W + b) ~ O (1) (SM4) and focus on either the semileptonic channel (11 + nj + T^t) or 
the dilcpton channel (2£ + nj + I$t) m the tf pair production (for a limit that takes into account the branching ratios 
in t' decay within the SM4, see [70]): 

pp -> i'F -> [V^+l , , . ,, t . b\W~], t . b^lubqqbll+l-vVbb (1) 

* ■* L J hadronic j Leptonic L J Leptonic x2 ' v ' 

The limits for the semileptonic channel were extracted using two dimensional fits to Mfu and Ht - the reconstructed 
mass of the t' and the scalar sum of pt of the visible jets and leptons and missing E T , respectively. In particular, 
the mass of the t' is reconstructed by fitting each event to the process pp — > W + W~bb — > Ivbqqb, i.e., taking the 4 
leading jets (with highest pt) and minimizing the x 2 for the equations: 

m (lv) = Mw 

m (qq) = M w (2) 
m (Ivb) = m (qqb) = Mfu 

On the other hand, the analysis for the dilepton channel relies on the fact that M^,, which is the invariant mass of a 
pair of any lepton and a 6-jet in the event, is much higher in the ft' signal with respect to the leading it background. 
In particular, in the case of ti, Ma, has an upper bound that corresponds to the mass of the top quark, and therefore 
in the region above 170 GeV ("signal region") Ma, is a clean signal for the t'. 

Below we will study the relevance of these limits in the general BSM4 case, in the feasible scenario where t' — > W + b 
is not the leading tf decay channel. This may occur in models with an extended Higgs sector, e.g. in the 4G2HDM 
suggested in [43]. In such models, the new scalars play a crucial role in the t' phenomenology, and new channels like 
t! — > ht, t' — > H + b (see also [38]) or even the SM4 forbidden t! — > W + b' may dominate. In [43 it was shown that 
all these channels are allowed by EWPD, and some of them dominate within large areas in the (EWPD allowed) 
parameter space. 
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Thus, in this scenario, pp —> Ivbqcjb / '£ + t~ vvbb are no longer the leading signatures for t' pair production at the 
LHC; the new channels can dominate, giving rise to several possible new signatures for t' at the LHC (see section 4). 
We argue that for these cases, the CMS fit to pp — > ivbqqb and pp — > t+l~vvbb should fail, as it attempts to impose 
these specific SM4-motivated dynamics on processes that have a completely different topology. In particular, for the 
scmilcptonic channel, taking only 4 leading jets out of all the jets in the event will miss a large part of the energy 
of the event and will, therefore, result in Mfn being substantially lower- exactly where the main ti background is. 
An example of that can be seen in Figure [T] for the 6W+2b signature which occurs in the 4G2HDM (see section 4) . 
For the dilepton channel, the analysis will fail for signatures with more than 2 leptons or 6-jets, as the combinatorial 
background will lower the Ma, (see section 6), resulting in much less events in the signal region. 

In the semileptonic channel the kinematics of each event is completely determined, thus we suggest using a recon- 
struction that does not assume a specific decay chain, and instead reconstructs any event of the type 1£ + nj + If? 
under the sole assumption that the lepton and the missing energy are the result of the leptonic decay of a W boson, 
similarly to [HD]. This will result in the correct reconstruction of t't' events (see section 5), regardless of the specific 
decay signature. 

The specifics of the such a reconstruction strategy are the following: we select all the jets with transverse momentum 
that is high enough, we then reconstruct the leptonic W and choose the correct partition of the event (which jet is 
originated from t' and which one from t' ) by minimizing the \ 2 of the following equations 

TO (W) = M w ^ 
m (Left Side) = m (Right Side) = M gen 

Another important point to notice is that for all the new signatures, the number of jets in the final state is higher 
than the SM background, and for some of the signatures there is also an excess of &-jets, as will be shown in section 
4. We will use cuts based on these properties in section 5, and, as we will further show, this will result in higher 
detection sensitivity for these signatures. 



III. THE 4G2HDM 



Let us recapitulate the underlying properties of the 4G2HDM, the full account of which appears in [43] . 
The 4G2HDM is a two-Higgs doublet model with 4 generations of chiral fermions, where the Yukawa terms are defined 
as follows: 

L Y = -Q L U t F (i - I^A + $ h F ■ J^M d R - Q L ($,G (I - 1^) + $ h G ■ I^A u R + h.c. (4) 



such that Q L is the left-handed SU(2) quark doublet, fum are the left (right)- handed fermion fields and F,G are 
arbitrary 4x4 Yukawa matrices in flavor space. Also, / is the identity matrix, I q ql3q are diagonal matrices defined as 
jag/3, _ ^ a g^ a q ,0 q ) and are the two Higgs doublets. 

The different choices of a q , /3 q will result in different Yukawa textures and, therfore, different models. In this work we 
assume (ay, /3 d ,a u , f3 u ) = (0, 1,0, 1) for which ^ couples only to the 4 th generations quarks, while couples to the 
first 3 generations. This setup was defined as 4G2HDM of type I in [43] and is partly motivated by the absence of 
(tree-level) FCNC in the first 3 generations. In particular, in this case, FCNC is only allowed between the third and 
fourth generations and it plays a crucial role in the phenomenology of the 4th generation fermions. 
The physical Higgs scalars that emerge after EWSB are: H ± (charged scalar), h, H (lighter and heavier CP-even 
neutral scalars) and A (CP-odd neutral scalar). Their couplings to the quarks are given by: 
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I (5) 
L(Aqiqj) = -i ,, q , q_i (m^tan^Sij - (tan/3 + cot/3) (m gi S?i? - m^S^L)) q 3 A, 

L(H + qiq,j) = -i Ui [m dj tan/3V Uidj - m dk (tan/3 + cot/3) V^Iljy] R+, 

[-m Uz tan(3V Uidj + m Uk (tan/3 + cot/3)Y, u k *V kj ] L}d J H+ , 

where q = u, d, I u . I d = +|, — |, V is 4x4 CKM matrix, a is the mixing angle in the CP even neutral Higgs sector and 
tan P is the ratio between the VEV of §h and the VEV of i.e. tan (3 — Also, S 9 are new mixing matrices in the 
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FIG. 2: Rag = r( ^" )4 T gCM as a function of a for m t > = m v = 450 GeV and tan/3 = 1. 
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g , can be parametrized in the following way 
\ 



(6) 
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where tb and et effectively parametrize the mixing between the third and fourth generation quarks in the down and 
up sectors, respectively. In the following we will set = and et — ^y, as natural values representative for these 
quantities (see [12]). 

IV. BEYOND SM4 DECAY PATTERNS OF t' AND &': A 4G2HDM CASE 



As mentioned above, in the 4G2HDM the 4 th generation quarks have a more complex decay pattern that stems from 
their interactions with the extended Higgs sector. In addition, the SM4 forbidden channels, t' — > Wb' and b' — > Wt' , 
are no longer in contradiction with the EWPD, and may be kinematically open as well. For simplification, we will 
assume that h is the only neutral scalar light enough to take part in the decays of t' and b'. We will thus consider 
the following decay modes: 

1. t' -> ht (&' ->• Kb). 

2. H -> H+b (6' -> H~t). 

3. t' -> Wb (b' -+ Wt). 

4. t' -> Wb' (6' -> W). 

We assume the decay mode if — > H + b' to be kinematically closed taking Mjj+ to be larger than the mass gap in the 
fourth generation doublet. 

To complete the picture of t' and b' decays, we need to fully account for the light neutral and charged Higgs decays. 
For h we have: 

1. h bb. This decay mode is prominent for Mh < 130 GeV. 

2. h — > W + W~ , ZZ. These modes can be suppressed in this model (as in any 2HDM model), due to the sin (a — (3) 
factor in the hWW and hZZ vertices. 
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3. h —> gg/"fj- Both channels are loop induced. In the SM4, h — > gg is the leading decay channel for Mh < 150 
GeV and is about 10 times larger than the SM, while h — > 77 is about 10 times smaller than the SM value 61]. 
On the other hand, in the 4G2HDM, h — > gg could be suppressed in some areas of the parameter space, as a 
result of a destructive interference between the heavy fermion loops due to extra factors in the corresponding 
Yukawa terms which depend on the angles a and (3 (see Eq. [5]). This would lead to either a suppression or 
enhancement of light Higgs production by gluon fusion as a function of a, as is demonstrated in Figure [2] It is 
obvious that, even for the "popular" choice of a = ft, the enhancement of Higgs production is only by a factor 
of 2, which is almost 5 times smaller than the corresponding enhancement in the SM4. This is an example of 
how an extended Higgs sector may suppress the Higgs production signal (see also [SHITT]), providing a possible 
explanation of the absence of an observed signal |58j . Similarly, h — ¥ 77 could be enhanced or suppressed in 
the 4G2HDM with respect to the SM4 or the SM [7TJ [75] . These issues are beyond the scope of this work and 
therefore will not be discussed here. 

4. h — > v T iv T i /t't' fit. For the purposes of this work, we assume that < 1m v ^ j1m T i j1m t allowing us to ignore 
these channels. 

The H + decays are the following: 

1. H+ ->■ tb 

2. H+ -> H b 

3. H+ -> hW+ 

4. H + -> tv t and H + -> rV T /. 

The decay modes for H + that involve the first two generations are much smaller than the above ones, due their light 
masses. Combining all these channels to a cascade decay, we are left with several topologies for the t' signatures, with 
different decay chains for each possibility: 

1. t' -> W + b (SM4-like) 

2. t' ->• W + 3b: 

. f->M-> [bb] h [W+b] t 

• t'^H+b^ [tb\ H+ b^ [{W+b) t b\ H+ b 

• t' -^W + b' -> W+ [hb] b , -> W+ [(66) h 6] 

3. t' ->• 3W + b 

• t' ->■ ht -> [W+ W~] h [W + b] t 

• t'-> H+b^ [hW+] H+ b^ [{W + W-) h W+] H+ b 

• *'->■ W + b' -> W+ [hb] b , -> W + [{w + w~) h b] b , 

While for b' decays we have: 

1. y -> 36 

• b' -> hb -> [66] h b 

2. 6' -> 2W + b (SM4-like) 

• b 1 -> hb^ [W+W~] h b 

• b' -> W~i -> [W+&] t W~ (SM4- Like) 

• 6' -> W"*' -> [W+6] t , W~ 

3. V — > 4VF + b 

. 6' H-i [W+6] t [(W+W-),, W-] H _ [W+b] t 

• t'W~ ->■ [ht] v W~ ->• [{W+W-) h {W + b) t ] t , W~ 

• b' -> W~ -> [# + 6] t , W~ -> [(/iVF+) ff+ &] t , W~ -> [({VF+PF-} h VF + ) ff+ &] t , TV" 



G 



4. b' -> 2W + 36 



• b' -> -> [/iW~] H _ [W+6] f -> [(bb) h W~] 



[W+b] t 



• b' -> t'W- -> [ht] t , W~ -> [(66) ft (W+6) t ] t , W~ 

• 6' -> iW- -> [i/+6] t , W~ -> [(/iW+) H+ 6] t< W~ 

• 6' ->■ t'W" ->■ [i?+6] t , ->■ [(t6) ff+ 6] t , W~ ->• 



[({^ + &} t 5) ff+ ^] ( , w- 



The decay chains not listed here, e.g. decay chains with Z bosons or fourth generation leptons, may further 
complicate the phenomenology of t' and b' and will not be considered in this work. 

The above t' and b' decay channels give rise to a set of new signatures for t' and b' pair production at the LHC 
which we will generically denote by pp — > t't' /b'b' — > nyyW + n^b, with nyy and being the number of W and b jets 
in the event, respectively. In particular, the t't' signatures are 2W+2b (SM4- like), 2W+6b, 6W+2b, 2W+4b, AW+2b 
and AW+Ab. As was mentioned earlier, for these signatures there is an excess of jets, leptons and 6-jets with respect 
to the SM and the SM4 signals, as is demonstrated in Figure [3] 

We stress again that, in general, these signatures are not exclusive to the 4G2HDM; they can appear in any model 
consisting of new (neutral or charged) scalars in the reasonable scenario that they interact mostly with the 3 rd and 
4 th generation fermions. 



For the semileptonic channel (pp —> t't' —tl£ + nj + I$t), we suggest a general reconstruction method that is 
based on [50] and test the extent to which the new signatures are already excluded by the "old" CMS analysis (which 
gives my > 450 GeV for the SM4) [52]. We focus our study bellow on the new pp -> t'W -> 2W + 66 or 6W + 2b 
signals, and show that, in those cases, t' can be detected at the LHC using the excess of jets and 6-jets mentioned 
above. The analysis is performed for two choices of parameter space in the 4G2HDM for which Br (t' — > th) ~ O (1), 
but arguably, due to the fact that no care is given to the specifics of each decay chain, our conclusions hold for any 
other model or parameter space that results in the same signature (in the previous section, we showed that for each 
signature there are several decay chains in the 4G2HDM with different particles in the intermediate states but with 
the same final state). 

We use Madgraph/MadEvent [55] to generate the signal and two J Ldt = 1 fb _1 sets of background events: W+jets 
and tt+jets, with if -factors of 1.5 for the signal and the tt+jets background [53], and 1.3 for W+jets background 
[55] , We use MLM parton-jet matching method [64] for the background with pTmin = 100 GeV for the tt+jets and 
PTmin = 150 GeV for TV+jets (see [63] ). We use BRIDGE [65 for the decay of the new particles and Pythia [66] for 
the decay of the SM particles, shower, fragmentation and hadronizations. We use PGS [67 with the LHC card for 
the detector simulation. 

We look for l£ + nj + T^t signatures, requiring exactly one muon or electron with px > 20 GeV, Ifr > 20 GeV, and 
at least four jets with p T > 120, 90, 35, 35 GeV and \r]\ < 2.5 

For the signal, we choose nty — 350,400,450 GeV, tan/3 = 1 , V34 = 0.1, and two different values for the Higgs 
mass, M h = 130, 170 GeVW , while the other scalars are assumed to be heavier and thus play no role in the decays. 
For both values of Mh the main decay mode is t' — > ht with a branching ratio of ~ 0.9. For Mh = 170 GeV, h decays 
mainly to W + W~ , while h —> bb for Mh = 130 GeV; this leaves us with the two aforementioned signatures: 6W + 2b 
and 2W + 66, respectively. First we demonstrate in Figure [3] the expected excess of jets and 6-jets for the 6b+2W 
signature. We would not use the jet and 6-jet distribution as a signal, as that might underestimate the background, 
especially the high multiplicity one (tt + nj with n > 2). Instead, we use the demonstrated excess as a basis for one of 
the cuts in the selection of the signal. When enough low multiplicity background events survive the cut, the addition 
of the high multiplicity background events will not play a dramatic role in the S/B ratio and will not change our 
conclusions. 

For the same signature, we compare Mm Jthe reconstructed mass with the SM4 hypothesis) and M gen (the general 
reconstruction mentioned earlier, see Eqn. [3| in Figure |4j while in Figure [5] we plot the distribution of M gen for the 
signal (2VF+66) and background events. We can see in Figure E] that the peak of the distribution of M gen is roughly 



[1] Although Mh m 12bGeV seems more realistic in view of the recent hints from the LHC, we fix = 130, 170 GeV for demonstration 



V. 



RESULTS: SEMILEPTONIC CHANNEL 



purposes. 
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FIG. 3: Number of jets (left) and number of 6-jets (right) histograms for thepp — s> t't' 2W + 6b signature (red) with rrty = 450 
GeV. Also shown are the backgrounds from VK+jets (green) and tt+jets (blue) for a set of 7 TeV LHC events with J Ldt — 1 
fb _1 , in the semileptonic channel (11 + nj + Lfr)- 




FIG. 4: Comparison between M/u = m (lub) = m (qqb) (the reconstructed t' mass using the CMS method, see Eq. [2| in 
blue and M gen — m (Left Side) — m (Right Side) (the reconstructed t' mass using our method, see Eq[3] ) in red, for the 
pp — > t't' — > 2W + 6b signature with m t i = 450 GeV at the LHC (7 Tev) with J Ldt — 1 fb _1 , in the semileptonic channel 
(11 + nj + L/it)- See also text. 

at nit' 5 while for Mfn the peak is misplaced, being substantially lower. We also note that the distribution of M gen 
in the new signatures is much broader around the peak than the distribution of Mfa; this is due to the massive 
combinatorial background coming from the high number of jets in the final state. In Figure [6] we plot the distribution 
of Ht for the signal and the background events for the same signature. 

Based on the above, we test the sensitivity of detection in our method for the two signatures (6W+2b and 2b+6W) 
for the 3 if mass values, m t ' — 350,400,450 GeV, using the following cuts: 

1. M gen > 300 GeV 

2. H T > 600 GeV 

3. Uj+bj = n-jets + Kb- jets > 6 

The results are summarized in Table [i] Wee see that the leading background is tt+jets, as VF+jets almost vanishes 
when imposing nj ets + n^—jets > 6. For m t i = 350 GeV we get a statistical significance of ^= ~ 4.7— 5.7, depending 
on the signature. For higher values of me the statistical significance drops, e.g., to the level of ~ 2.3 — 2.9 for the 




FIG. 5: M gen for pp — > t't' — > 2W + 6b with m t < = 450 GeV (red) together with VK+jets (green) and tt+jets backgrounds 
(blue) at the LHC (7 Tev) with j Ldt = 1 fb _1 , in the semileptonic channel (U + nj + I$t)- 
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MGeV] 

FIG. 6: Ht for pp — > t't' — > 2W + 6b with m t / = 450 GeV (red) together with VK+jets (green) and tt+jets backgrounds (blue) 
at the LHC (7 Tev) with j Ldt — 1 fb -1 , in the semileptonic channel (U + nj + I$t)- 

two signatures when m t > = 450 GeV . In contrast, applying the "standard" CMS method (i.e. using Mfu instead of 
M gen and requiring that at least one of the jets is tagged as a 6-jet instead of the nj + bj cut), we get ~ 2.6 for 

mt, = 350 GeV and -4= ~ 1.7 for m v = 450 GeV. 
v a 

We proceed to test the consistency of the CMS exclusions results in the semileptonic channel (which exclude t' — > Wb 
as is the case in the SM4) with the existence of a BSM4 signal in the current data. We recall that the CMS analysis 
is based on a 2d (Mf it and Ht) likelihood fit to the data under the hypothesis of signal (SM4) + background and the 
hypothesis of background only, and a CLs method [731 f° r the computation of the exclusion limits. They found 
that m t > > 450 GeV at 95% CL (for the SM4 case) [52]. 

We perform a simplified computation of what would the CMS exclusion limit be for m t i in the case of background 
only and in the case of background+BSM4. To do so, we generate 1000 LHC events (J Ldt = 1 fb -1 ) for each case. 
We then use the ratio L = Lb/L s+ b of the likelihoods to the signal (SM4) + background hypothesis (L s+ b) and to the 
background hypothesis (L{,), as the test statistic. We naively assign a 20% systematic uncertainty on the background 
cross-section, minimizing the likelihood with respect to this parameter in each case. We calculate the likelihood ration 
(L) for each simulated signature, and determine the probability to observe a higher value of L for the background only 
hypothesis (CLfc) and the signal+background hypothesis(CL s+ 6(cr)). The 95% CL limit on the signal cross-section is 
then the value of the signal cross-section er, for which 

CL S = ^±^1 = 0.05. (7) 
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VK+jets 




626 
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2W+66 m t > 


= 350 GeV 


172 


99 


93 


138 


56 


5.66 


2^+66 m t i 


= 400 GeV 


119 


74 


69 


104 


45 


4.55 


2W+66 m t > 


= 450 GeVe 


70 


46 


43 


65 


29 


2.93 


6W+2b m t > 


= 350 GeV 


168 


80 


87 


135 


46 


4.65 


6W+2b m t , 


= 400 GeV 


113 


58 


60 


99 


34 


3.43 


6W+26 m t , 


= 450 GeV 


77 


43 


40 


74 


23 


2.32 



TABLE I: Number of events after the selection (in the second column), after each of the three cuts: rij+bj = rij e ts + rib- jets > 6, 
M gen > 300 GeV , H T > 600 GeV (3 rd - 5 th columns) and all of them combined (6 th column). for each of the new 

signatures at the LHC (7 Tev) with J Ldt — 1 fb _1 is given in the seventh column. 




FIG. 7: The 95% CL exclusion plot distribution on the t' mass assuming the SM4 signature in the semileptonic channel 
(11 + nj + F£t) ■ For the case of background only, the red doted line is the median and the yellow and green bands are the ±1 
and ±2 standard deviations accordingly. The black line is the median for background + t't' — > 2b + 6W with m t i — 450 GeV 
and the blue line is the median for background + t't' — > 66 + 2W with m t i = 350 GeV. The results for the rest of the signatures 
with m t i = 350 — 450 GeV lie between these two lines. 



Finally, we calculate the distribution of the 95% CL limit on the signal cross-section for background only and for 
background+BSM4 hypothesis, and plot in Figure [7] the exclusion limits for background only, and for background 
with two chosen BSM4 signatures for t' pair production: t't' -t 2b + 6W with m t > = 450 GeV and t't' — !• 66 + 2W 
with m t i = 350 GeV. The exclusion limits for all the other BSM4 signatures that we've checked (6b+2W ,2W +6b with 
m t > = 350, 400, 450 GeV) are between these two limits for the chosen signatures. We observe that for ra v — 350 GeV, 
the median of the exclusion limit distribution for the BSM4 signatures is within 2 standard deviations of the expected 
limit for the background only case, and for m t ' = 450 GeV it is fully within 1 standard deviation of the expected limit 
for the background only case. Thus, the discrepancy between the CMS observed exclusion limit, which is close to the 
expected one, see [S5] (the expected limit in our naive calculation is denoted by the red dotted line in Figure [7]), and 
the aforementioned BSM4 scenarios is less than 2a for m t > ^ 350 GeV, and l-2a for m t > = 350 GeV. We, therefore, go 
on to conclude that BSM4 scenarios with m t i > 350 GeV, can be consistent with the current semileptonic data and 
therefore the limit on the t' mass for the BSM4 case should be around 350 GeV in the semileptonic channel. 
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M™ ,n [GeV] 



FIG. 8: Mft in for the SM4-like pp -t t't' -t 2W + 2b signature (red) and for the BSM4 pp -> t't' — s> 2W + 6b signature(blue) 
with m t i — 350 GeV for a set of 7 TeV LHC events with J Ldt = 5 fb _1 in the dilepton channel. The black line is approximately 
at the top mass and the region to the right of the line is the signal region. 

VI. RESULTS: THE DILEPTON CHANNEL 

The most stringent bounds to this date on the t' mass (557 GeV) are set by the CMS, analyzing the dilepton channel 
(pp — > t't' — > l + l~vVbb) for t' pair production and decay at the LHC [53] ■ The signal in this channel is relatively rare 
(about 11% of the t't' events) but clean due to the presence of the two leptons that suppresses the SM background. 
The suppression of the background is further promoted by requiring that exactly two of the jets in each selected event 
are tagged as 6-jets. The kinematic variable that is used to discriminate the signal from the huge tt background is the 
invariant mass of one of the leptons and one of the 6-jets, M,™ m , where the configuration with the lowest invariant 
mass is chosen. In the case of it production this invariant mass is bounded from above by the top mass (170 GeV), 
while for the t't' signal, there should be a large portion of events giving M™" > 170. This leaves the region above 
this mass (named the "Signal Region") almost background-free (except mostly misidentification background) and 
completely signal-dominated. The number of events in the Signal Region is therefore used for the limit calculation. 

When the new textures of t' decay with higher multiplicities of leptons and 6-jets are considered, the dominance 
of the signal for M™" > 170 GeV may no longer be realized, due to the combinatorial background and the altered 
kinematic shape of the events. We try to address this issue using a simulation similar to the one in the previous 
section, with different selection rules. We generate t't' events with 5 fb _1 for the SM4-like 2W+2b signature and for 
the new 2W+6b signature, choosing m t > — 350 GeV and selecting events with at least two leptons and exactly two 
6-jets, with the kinematic cuts that were used in the CMS analysis [53] . Due to the fact that the ^-tagging efficiency 
in PGS [57] is not well calibrated with the efficiency of the TCHEM method of 6-tagging [75] that was used in the 
CMS analysis, we get a lower number of events for the SM4-like signature than the number of events reported in [53] . 
We thus use a higher efficiency to roughly simulate the correct number of events in the Signal Region. Thus, our 
results below should be considered only as an illustration of the effect of BSM4. 

The results are presented in Figure [8] We first note, that the total number of selected events is lower for the 2W+6b 
than for the SM4 case. This is due to the fact that there is a higher number of b jets (6) in 2W+6b events, thus, 
for high enough 6-tagging efficiency (above 0.5) there is a lower chance of tagging exactly 2 of them. In the signal 
region, the number of the events for the 2W+6b signature is negligible- about 1/10 of the events in SM4 case. In 
fact, for m t > > 350 GeV, the number of 21^+66 events in the Signal Region is even lower due to the lower production 
cross-section. We therefore conclude that the current stringent limit on the t' mass (557 GeV) in the dilepton channel 
is completely irrelevant for some BSM4 scenarios, for > 350 GeV (we recall that my — 350 GeV is the lowest 
mass considered in [53]). 
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VII. SUMMARY 



We have explored new detection strategies for the t' that hold in the general case, where new decay patterns 
(beyond the SM4) for t' emerge, e.g. t' — > th. In such cases, new t' signals, such as pp — > ft' — > 6W + 26 and 
pp —> ft' — > 2W + 66 that we have studied here may dominate. As a representative framework for BSM4 we chose 
the 4G2HDM of which is a variation of a 2HDM framework with 4 generations. We then studied its implications 
for t' phenomenology at the LHC, keeping the analysis as general as possible and, therefore, model-independent to a 
large extent. 

We have shown that if the new f decay channels lead to pp — > ft' — > 2W + 66 or to pp — > ft' — > 6W + 26, then the 
current limits on the t' mass (450/557 GeV for the semileptonic and the dilepton channel in the SM4 case) do not 
apply and a different detection strategy is required. For the semileptonic channel, wc estimate the current limit on 
the t' mass in the BSM4 case to be around 350 GeV, demonstrating that a general reconstruction method (without 
the assumption of a specific topology of each event) is successful in pushing the limit upwards. 

In the dilepton channel, we find the CMS limit to be completely irrelevant for the mass range in the analysis 
(350-600 GeV) for several BSM4 scenarios. This includes the case that the mass of the lightest Higgs is around 130 
GeV, which may be consistent with what ATLAS and CMS see in the Higgs searches. We leave for a future study 
the constraints on new scenarios resulting from the 6/ searches at the LHC [551 [57] . 
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